Distributed processing systems

ABSTRACT

A distributed processing method is disclosed herein for evaluating student work product. The distributed processing system includes a server, a database server, and an application server that are interconnected via a network, and connected via the network to a plurality of independent processing units. The independent processing units can include an analysis engine that is machine learning capable, and thus uniquely completes its processing tasks. The server can provide one or several pieces of data to one or several of the independent processing units, can receive an analysis results from the one or several independent processing units, and can update the result based on a value scoring the machine learning of the independent processing unit.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/606,973, entitled DISTRIBUTED PROCESSING SYSTEMS, filed on May 26,2017, which is a continuation of U.S. patent application Ser. No.14/462,451, entitled DISTRIBUTED PROCESSING SYSTEMS, filed on Aug. 18,2014, which claims the benefit of U.S. Provisional Application No.61/866,990, filed on Aug. 16, 2013, and U.S. Provisional Application No.61/867,531, filed Aug. 19, 2013, the entirety of each of which arehereby incorporated by reference herein.

BACKGROUND OF THE INVENTION

A distributed system is a software system in which components located onnetworked computers communicate and coordinate their actions by passingmessages. The components interact with each other in order to achieve acommon goal.

Use of a distributed system to perform distributed processing providesmany benefits in that distributed processing can efficiently combine andutilize processing resources at a variety of locations to quickly solvelarge and/or complicated problems. In one example of this, a group ofnetworked computers may be controlled by a group of users. However, asnormal use of a computer does not always use the computer's entireprocessing capacity, each of the computers in this group of computersmay have some latent, or unused processing capacity. This latent orunused processing capacity can be identified and combined together tosolve another problem. Thus, use of the processing capacity of thecomputers of the group of computers is maximized, which maximization canlead to significant infrastructure savings associated with acquisitionof otherwise needed processing capacity.

However, distributed processing still has many shortcomings which limitits usefulness. Thus, new methods, techniques, and systems are requiredto improve distributed processing.

BRIEF SUMMARY OF THE INVENTION

One aspect of the present disclosure relates to a distributed processingnetwork. This distributed processing network can improve processingspeeds. The network can include a source device that can provide groupsof data, each of which groups of data can be associated with one orseveral user authors, and which groups of data together can form aprocessing task, a plurality of independent processing units that canreceive a portion of the processing task, which portion of theprocessing task can include one or several of the groups of data, andthat can characterize one or several aspects of the one or several ofthe groups of data. In some embodiments, each of the plurality ofindependent processing units can include a standards module that canreceive and store an analysis criterion and an analysis guideline, whichanalysis criteria identifies one or several attributes for identifying acharacterization of the one or several of the groups of data, and whichanalysis guideline identifies one or several rules for application ofthe analysis criteria, and an analysis module that can provide acharacterization of the one or several of the groups of data accordingto the analysis criteria and the analysis guideline. The distributedprocessing system can include a server communicatingly connected to thesource device and the plurality of independent processing units via anetwork. The server can be able to receive a signal encoding theprocessing task, separate the one or several groups of data from theencoded processing task, generate a signal encoding a first one of theone or several groups of data, send the signal encoding the first one ofthe one or several groups of data to a first one of the plurality ofindependent processing units and to a second one of the plurality ofindependent processing units, receive a first electric signal from thefirst one of the plurality of independent processing units, which firstelectric signal can be an encoded first characterization generated bythe first one of the plurality of independent processing units of thefirst one of the one or several groups of data, receive a secondelectric signal from the second one of the plurality of independentprocessing units, which second electric signal can be an encoded secondcharacterization generated by the second one of the plurality ofindependent processing units of the first one of the one or severalgroups of data, retrieve an accuracy parameter, which accuracy parametercontains a first component associated with the first one of theplurality of independent processing units and a second componentassociated with the second one of the plurality of independentprocessing units, which first component can include a norming value thatstandardizes the first characterization when applied thereto, and whichsecond component can include a norming value that standardizes thesecond characterization when applied thereto, and calculate a combinedcharacterization of the first one of the one or several groups of databased on a combination of the first and second characterizations and theaccuracy parameter.

In some embodiments, the server can further select a group of theplurality of independent processing units for processing of theprocessing task. In some embodiments, the server can include a storedmanagement guideline that includes one or several rules for thecompletion of the analysis of the processing task. In some embodiments,the management guideline specifies a number of reviews for one orseveral of the one or several groups of data in the processing task.

In some embodiments, at least one of the independent processing units isa computer. In some embodiments, the server can increment a count eachtime that the first one of the one or several groups of data is sent toone of the plurality of independent processing units, and compare thecount to a threshold value to determine whether to send the first one ofthe one or several groups of data to an additional one of the pluralityof independent processing units. In some embodiments, the server canselect an additional one of the plurality of independent processingunits if it is determined to send the first one of the one or severalgroups of data to an additional one of the plurality of independentprocessing units. In some embodiments, the server can receive anadditional electric signal from the additional one of the plurality ofindependent processing units, which first electric signal can be anencoded first characterization generated by the additional one of theplurality of independent processing units of the first one of the one orseveral groups of data.

In some embodiments, the server can update the accuracy parameter withrespect to the first one of the plurality of independent processingunits. This updating can include comparing the combined characterizationof the first one of the one or several groups of data to the firstcharacterization and the accuracy parameter, and incrementing ordecrementing the first component of the accuracy parameter based on thecomparison. In some embodiments, updating the accuracy parameter furtherincludes retrieving an index function, which index function generates anaccuracy parameter update value for the first component of the accuracyparameter according to the difference between the first characterizationand the combined characterization. In some embodiments, the firstcomponent of the accuracy parameter is incremented or decremented by theaccuracy parameter update value.

One aspect of the present disclosure relates to a distributed processingnetwork. The distributed processing network can include features and/orfunctionalities that improve processing speeds. The network includes asource device that can provide groups of data, each of which groups ofdata is associated with one or several user authors, and which groups ofdata together form a processing task. In some embodiments, thedistributed processing network can include a plurality of independentprocessing units that can receive a portion of the processing task,which portion of the processing task can include two or more of thegroups of data, and the independent processing units can relatively rankthe two or more of the groups of data according to one or severalaspects of the two or more of the groups of data, each of whichplurality of independent processing units can include a standards modulethat can receive and store an analysis criterion and an analysisguideline, which analysis criterion identifies one or several attributesfor relatively ranking two or more of the groups of data, and whichanalysis guideline identifies one or several rules for application ofthe analysis guideline, and an analysis module that can provide arelative ranking of the two or more of the groups of data according tothe analysis criteria and the analysis guideline. The independentprocessing network can include a server communicatingly connected to thesource device and the plurality of independent processing units via anetwork. The server can receive a signal encoding the processing task,separate the two or more groups of data from the encoded processingtask, determine a number of relative rankings to complete analysis ofthe processing task, generate a plurality of pairs to fulfill the numberof relative rankings to complete analysis of the processing task, eachof the plurality of pairs having at least two of the two or more groupsof data, select a set of the plurality of independent processing units,which independent processing units in the set of the plurality ofindependent processing units are designated to relatively rank at leastone pair of the plurality of pairs, and which one pair of the pluralityof pairs is relatively ranked by identifying which of the at least twoof the two or more groups of data is better based on the one or severalattributes of the analysis criteria, generate a plurality of signalsencoding the plurality of pairs, send signals encoding the plurality ofpairs to the set of independent processing units. In some embodiments, asignal can be sent to each of the set of independent processing unitsidentifying at least one pair of the plurality of pairs. The server canreceive signals from the independent processing units in the set ofindependent processing units, which signals contain the relativerankings of the two of the two or more groups of data forming the pairsof the plurality of pairs, identify a bounding piece that can be one ofthe groups of the two or more groups of data, and calculate acharacterization of the groups of the processing task based on thebounding piece and the ranked sequence.

In some embodiments, the server can generate a ranked sequence of thetwo or more groups of data. In some embodiments, the bounding piece canbe one of the groups of the plurality of the groups of data and anassociated characterization. In some embodiments, the server canretrieve the bounding piece from memory. In some embodiments, the servercan control the generation of the bounding piece.

In some embodiments, the server can select one or several of theplurality of independent processing units for characterization of apotential bounding piece, which one or several of the plurality ofindependent processing units are selected based on an accuracy parameterretrieved by the server, which accuracy parameter identifies the degreeto which one or several of the plurality of independent processing unitshave accurately characterized one or several previous data groups. Insome embodiments, the server can identify one of the groups of the twoor more groups of data as the potential bounding piece, generate arequest signal encoding the bounding piece, send the request signalencoding the bounding piece to the selected one or several of theplurality of independent processing units, receive a first signal fromthe selected one or several of the plurality of independent processingunits, which first signal can include an encoded characterization of thepotential bounding piece generated by the selected one or several of theplurality of independent processing units, and store an indicatoridentifying the potential bounding piece as a bounding piece and storingthe characterization of the bounding piece.

In some embodiments, the bounding piece can be a plurality of boundingpieces, and in some embodiments, some of the plurality of boundingpieces can have different characterizations. In some embodiments, atleast one of the independent processing units can be a computer.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description provided hereinafter. It shouldbe understood that the detailed description and specific examples, whileindicating various embodiments, are intended for purposes ofillustration only and are not intended to necessarily limit the scope ofthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of one embodiment of a distributedprocessing system.

FIG. 2 is a schematic illustration of one embodiment of an independentprocessing unit of a distributed processing system.

FIG. 3 is a schematic illustration of one embodiment of a databaseserver of a distributed processing system.

FIG. 4 is a schematic illustration of one embodiment of the server ofone embodiment of a distributed processing system.

FIG. 5 is a swim-lane diagram illustrating one embodiment of a processfor increasing the efficiency and processing speed of distributedcomputing.

FIG. 6 is a flowchart illustrating one embodiment of a process forcalculating an actual score with a distributed processing system.

FIG. 7 is a flowchart illustrating one embodiment of a process forcalculating an accuracy parameter with a distributed processing system.

FIG. 8 is a flowchart illustrating one embodiment of a process foranalyzing a piece of data and/or a group of data with a distributedprocessing system.

FIG. 9 is a flowchart illustrating one embodiment of a process forgenerating the evaluation with a distributed processing system.

FIG. 10 is a flowchart illustrating one embodiment of a process forparallelized evaluation with a distributed processing system.

FIG. 11 is a flowchart illustrating one embodiment of a process foranalyzing a piece of data and/or a group of data with a distributedprocessing system.

FIG. 12 is a flowchart illustrating one embodiment of a process forgenerating the relative ranking with a distributed processing system.

FIG. 13 is a block diagram of an embodiment of a computer system.

FIG. 14 is a block diagram of an embodiment of a special-purposecomputer system.

In the appended figures, similar components and/or features may have thesame reference label. Where the reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same reference label. Further, various componentsof the same type may be distinguished by following the reference labelby a dash and a second label that distinguishes among the similarcomponents. If only the first reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same first reference label irrespective of thesecond reference label.

DETAILED DESCRIPTION OF THE INVENTION

The present disclosure relates to systems, methods, and devices forincreasing the efficiency and processing speed of distributed computing.Particularly, in some embodiments, distributed computing can be used toevaluate and/or analyze a plurality of unique pieces of data or datagroups. In some embodiments, the number of pieces of data or data groupsmay be large, and the processing times for some or all of the pieces ofdata and/or data groups can be long. Because of this, significantprocessing resources can be required for completion of a processingtask, which can include a plurality of the pieces of data and/or datagroups.

In some embodiments, the processing can be performed by one or severalindependent processing units which can, in some embodiments, beintelligent in that they are able to learn and adapt based on externalstimulus. Particularly, the one or several independent processing unitscan learn and thereby improve the quality of their output as theyevaluate one or several of the pieces of data. While there are manybenefits of such a system of intelligent independent processing units,such a system is disadvantaged in that the evaluation of the independentprocessing unit is unique as it is based on the learning, which in manycases is also unique, of that independent processing unit. Further, inmany instances, the uniqueness of the evaluation of the independentprocessing unit is based on so many variables, many of which areunknown, that it is difficult to identify the source of the uniqueness,and thereby seek to eliminate the uniqueness.

Instead of attempting to identify a source of uniqueness, in someembodiments, the uniqueness of the evaluations can be eliminated and/ormitigated through one or several standardization processes and systems.

In one embodiment, these processes and systems can utilize one orseveral independent processing units to provide multiple evaluations forthe same piece of data and/or data group. These evaluations can becombined, and, in some embodiments, combined according to an accuracyparameter that is associated with one or several independent processingunits, to generate a combined evaluation. This accuracy parameter can bebased on data relating to the degree of difference between one or moreprevious evaluations of one or more pieces of data and/or data groups byone or more independent processing units and one or more finalevaluations for each of the one or more pieces of data and/or datagroups.

In another embodiment, the processes and systems can simplify theprocessing task to mitigate the adverse effect of uniqueness on theevaluation of one or more pieces of data and/or data groups. In such anembodiment, pieces of data and/or data groups can be associated in, forexample, groups of 2 (pairs), groups of 3, groups of 4, groups of 5,groups of 10, and/or any other or intermediate value. These associationsof pieces of data and/or groups of data can be provided to one orseveral independent processing units, which independent processing unitscan relatively rank the pieces of data and/or groups of data from bestto worst. The relative rankings can then be used to assign a finalevaluation to one or several of the pieces of data and/or groups ofdata.

Advantageously, both of the above mentioned embodiments increase theeffective processing speed with which the processing task is completed,additionally, as both of these mentioned embodiments decrease the totalnumber of evaluations of each of the pieces of data and/or groups ofdata used to generate a final evaluation. Additionally, by identifyingthe accuracy parameter, the distributed processing system is able todirect the pieces of data and/or groups of data to the independentprocessor best suited for generation of the evaluation.

With reference now to FIG. 1, a schematic illustration of one embodimentof a distributed processing system 100, also referred to herein as adistributed processing network, is shown. The distributed processingsystem 100 can include a main server 102, also referred to herein as theserver. The server 102 can be any desired type of server including, forexample, a rack server, a tower server, a miniature server, a bladeserver, a mini rack server, a mobile server, an ultra-dense server, asuper server, or the like. The server 102 can include any desiredhardware components including, for example, a motherboard, a processor,memory, one or several hard drives, a network connection, a powersupply, and/or the like. In some embodiments, the server can include auser interface that can be configured to provide information to a userand to receive inputs from a user. In one embodiment, the user interfacecomprises a screen having a plurality of pixels that can be controlledto transform data from electronic format to one or several characters,symbols, text strings, or the like. In some embodiments, the server 102can provide instructions to and receive information from the othercomponents of the distributed processing system 100. The server 102 canact according to stored instructions, which stored instructions can belocated in memory associated with the server 102 and/or in othercomponents of the distributed processing system 100. In someembodiments, the processor in the server 102 can comprise amicroprocessor, such as a microprocessor from Intel® or Advanced MicroDevices, Inc.®, or the like.

In some embodiments, and as depicted in FIG. 1, the server 102 canconnect to a network 104. The network 104 allows communication betweenthe components of the distributed processing system 100. The network 104can be, for example, a local area network (LAN), a wide area network(WAN), a wired network, wireless network, a telephone network such as,for example, a cellphone network, the Internet, the World Wide Web, orany other desired network. In some embodiments, the network 104 can useany desired communication and/or network protocols.

As further depicted in FIG. 1, the distributed processing system 100 caninclude a source device 106. In some embodiments, the source device 106can be any device that can receive one or several data pieces and/ordata groups. In some embodiments, the source device 106 can be a server,a computer, a tablet, a smartphone, a modem, a fax machine, a scanner,or the like. In some embodiments, the source device 106 can be integralto the server 102, and in some embodiments, the source device 106 can bedistinct from the server 102.

In one embodiment, the source device 106 can be communicatinglyconnected to one or several originating devices, which originatingdevices create the one or several pieces of data and/or groups of data.In some embodiments, the originating devices can be a device that canreceive one or several inputs such as, for example, inputs from asensor, user inputs, or the like. In one embodiment, an originatingdevice can include a computer that is receiving one or several inputsfrom a user, such as, for example, one or several typed text stringsfrom the user. The originating device can, in some embodiments,transform the received inputs, which can be, for example, one or severaltext strings into an electric signal.

The distributed processing system 100 can include a database server 108.In some embodiments, the database server 108 can be embodied in hardwareor in software and can provide database services to the other componentsof the distributed processing system 100. The database server 108 can,in some embodiments, be integral to the server 102, and in someembodiments, the database server 108 can be distinct from the server102. The database server 108 can store the one or several pieces of dataand/or groups of data that are received by the source device 106, andcan additionally store information relating to other components of thedistributed processing system 100, including, for example, thefunctionality, accuracy, and/or efficiency of other components of thedistributed processing system 100. The database server 108 can includeone or several databases which will be discussed at greater lengthbelow.

In some embodiments, the distributed processing system 100 can includean application server 104. The application server 110 can be embodied inhardware or in software and can facilitate in the control and managementof application operations between one or several users and the server102 and/or the server database 108. The application server 110 can, insome embodiments, be integral to the server 102, and in someembodiments, the application server 110 can be distinct from the server102.

The distributed processing system 100 can include a plurality ofindependent processing units 112. The plurality of independentprocessing units 112 can be one or several independent processing units114, shown specifically as a first independent processing unit 114-A, asecond independent processing unit 114-B, a third independent processingunit 114-C, a fourth independent processing unit 114-D, and a fifthindependent processing unit 114-E. In some embodiments, the independentprocessing units 114 can each be a computing device comprising aprocessor, which can comprise a microprocessor, such as a microprocessorfrom Intel® or Advanced Micro Devices, Inc.®, or the like, memory, auser interface configured to provide information to a user and toreceive one or several inputs from a user, or the like. In someembodiments, the independent processing units 114 can be computers.

In some embodiments, the independent processing units 114 can beconfigured to receive one or several data pieces and/or groups of datafrom the server 102 or other component of the distributed processingsystem 100 and to evaluate the one or several data pieces and/or groupsof data. In some embodiments, this evaluation can include a relativeranking of the one or more pieces of data and/or the groups of data. Insome embodiments, this evaluation can include a characterization of thepieces of data and/or groups of data according to one or severalcriteria and/or guidelines.

As used herein, an “evaluation” refers to the result of an analysis ofone or several pieces of data and/or groups of data that include both acharacterization and an explanation of the characterization. As usedherein, a “relative ranking” refers to the identification of therelative quality of one or several of the one or several pieces of dataand/or groups of data as compared to each other. As used herein, a“characterization” refers to an indicator of the result of theevaluation. This indicator can be a numeric value, an alphabeticcharacter, an alphanumeric character, any other symbol, or the like.

As used herein, the “explanation of the characterization” can identifyone or several factors leading to the characterization. These factorscan be identified in an analysis criterion that identifies one orseveral analysis parameters, the degree to which they are triggered,and/or the frequency with which they are triggered in the one or severalpieces of data and/or groups of data. In some embodiments, these factorscan be used to determine a parameter value. In some embodiments, theaggregate of the parameter values of an analysis criterion determinesthe characterization of the one or several pieces of data and/or groupsof data. The analysis parameters can, in some embodiments, describe oneor several attributes of the one or several pieces of data and/or datagroups. In some embodiments, one or several of these attributes can becomputer-detectable. In some embodiments, the degree to which theattributes occur, and/or the frequency with which they occur influencesthe degree to which, and/or the frequency with which, the analysisparameters are triggered, and thereby influences the parameter value.

In some embodiments, the analysis criteria can be unique and/or specificto each processing task comprising a plurality of pieces of data and/orgroups of data, and in some embodiments, the analysis criteria can benon-unique to each processing task. The analysis criteria can be storedin memory of the distributed processing system 100 such as, for example,the server 102 and/or the database server 108. In some embodiments, theanalysis criteria can be received from the source device 106; in someembodiments, the analysis criteria can be received from a user deviceconnected with the server 102; and in some embodiments, the analysiscriteria can be directly input into the server 102.

In some embodiments, the analysis criteria can be applied according toan analysis guideline. In some embodiments, the analysis guidelineidentifies one or several rules for application of the analysiscriteria. The analysis guideline can increase the uniformity acrossevaluations. In some embodiments, the analysis guideline can be uniqueand/or specific to each processing task comprising a plurality of piecesof data and/or groups of data, and in some embodiments, the analysisguideline can be non-unique to each processing task. The analysisguideline can be stored in memory of the distributed processing system100 such as, for example, the server 102 and/or the database server 108.In some embodiments, the analysis guideline can be received from thesource device 106; in some embodiments, the analysis guideline can bereceived from a user device connected with the server 102; and in someembodiments, the analysis guideline can be directly input into theserver 102.

In some embodiments, the distributed processing system 100 can beaccessed via one or several devices. These devices, also referred toherein as user devices, can comprise any computing device and caninclude, for example, a computer, a tablet, a smartphone, or the like.In some embodiments, these devices may be used by any individualincluding, for example, an employer, an employee, an administrator, asupervisor, an educator, a parent, a student, a trainer, a trainee, orany other individual or group of individuals.

With reference now to FIG. 2, a schematic illustration of one embodimentof the independent processing unit 114 is shown. As discussed above, theindependent processing unit 114 can be configured to provide informationto and/or receive information from other components of the distributedprocessing system 100. The independent processing unit 114 can accessthe distributed processing system 100 through any desired means ortechnology, including, for example, a webpage, a web portal, or vianetwork 104. As depicted in FIG. 2, the independent processing unit 114can include a network interface 200. The network interface 200 allowsthe independent processing unit 114 to access the other components ofthe distributed processing system 100, and specifically allows theindependent processing unit 114 to access the network 104 of thedistributed processing system 100. The network interface 200 can includeone or several features configured to send and receive information,including, for example, an antenna, a modem, a transmitter, a receiver,or any other feature that can send and receive information. The networkinterface 200 can communicate via telephone, cable, fiber-optic, or anyother wired communication network. In some embodiments, the networkinterface 200 can communicate via cellular networks, WLAN networks, orany other wireless network.

The independent processing unit 114 can include a user interface 202that communicates information to, and receives inputs from, a user. Theuser interface 202 can include a screen having one or several pixels, aspeaker, a monitor, a keyboard, a microphone, a mouse, a touchpad, akeypad, or any other feature or features that can receive inputs from auser and provide information to a user. In some embodiments, the pixelsof the screen can be controlled to provide information to the user ofthe independent processing unit 114. In some embodiments, the userinterface 202 can allow the user to provide one or several inputs to theindependent processing unit 114 that are relevant to the othercomponents of the distributed processing system 100 discussed herein.

The independent processing unit 114 can include an analysis engine 204.In some embodiments, the analysis engine 204 can be configured toreceive one or several pieces of data and/or groups of data from theserver 102, and/or from the database server 108. In some embodiments,the analysis engine 204 can be an embodiment in hardware or softwarethat can be configured to retrieve the analysis criteria and theanalysis guideline. In some embodiments, the analysis engine 204 can beconfigured to learn and adapt based on external stimulus, andparticularly based on the one or several of the pieces of data or datagroups previously evaluated by the analysis engine 204. In someembodiments, the analysis engine can be configured to generate a userinterface that can facilitate the human review of the one or severalpieces of data and/or data groups.

The analysis engine 204 can generate the evaluation according to theanalysis criteria and the analysis guideline. In such an embodiment, theanalysis module 204 can identify one or several analysis parameters ofthe analysis criteria, can determine the degree to which the attributesassociated with the analysis parameters occur, and/or can determine thefrequency with which the attributes occur in the one or several piecesof data and/or groups of data. These attributes can relate to, forexample, word usage, vocabulary, spelling, grammar, content, citations,use of citations, or the like. In one embodiment, for example, thesecriteria can include the number and/or frequency of misspellings, thenumber and/or frequency of grammar errors, the level of vocabulary ofthe one or several pieces of data and/or data groups, the correctnessand/or level of word usage, the coherency, accuracy, and/or relevancy ofthe pieces of data and/or data groups, or the like.

The independent processing unit 114 can include a standards engine 206,also referred to herein as a standards module. The standards engine 206can be embodied in hardware or software. In some embodiments, thestandards engine 206 can be configured to receive and store the analysiscriteria and the analysis guideline. In some embodiments, the standardsengine 206 can receive the analysis criteria and the analysis guidelinefrom another component of the distributed processing system 100 such as,for example, the server 102 and/or the database server 106. Thestandards engine 206 can be further configured to maintain the operationof the analysis module 204 in accordance with the analysis criteria andthe analysis guideline. In some embodiments, the standards engine 206can be configured to convert all or portions of the analysis criteriaand/or analysis guidelines to visual format and to control the userinterface 202 to display these. In some embodiments, the standardsengine 206 can be configured to convert the analysis criteria and/oranalysis guideline from the form in which they were provided to thedistributed processing system 100 to one or several lines of computercode that can control the operation of the analysis engine 204.

With reference now to FIG. 3, a schematic illustration of one embodimentof the database server 108 is shown. The database server 108 can includea network interface 300. The network interface 300 allows the databaseserver 108 to access the other components of the distributed processingsystem 100, and specifically allows the database server 108 to accessthe network 104 of the distributed processing system 100. The databaseserver 108 can include features configured to send and receiveinformation, including, for example, an antenna, a modem, a transmitter,receiver, or any other feature that can send and receive information.The network interface 300 can communicate via telephone, cable,fiber-optic, or any other wired communication network. In someembodiments, the network interface 300 can communicate via cellularnetworks, WLAN networks, or any other wireless network.

The database server 108 can include one or several databases. Thesedatabases can store information and/or data relevant to the functions ofthe distributed processing system 100. In some embodiments, thesedatabases can contain, for example, the one or several pieces of dataand/or data groups, the analysis criteria, the analysis guideline, oneor several accuracy parameters, previous evaluations, informationrelating to the originating devices and/or the sources of the one orseveral pieces of data and/or groups of data, or the like.

In one embodiment, these databases can include, for example, a sourcedatabase 302. The source database 302 can comprise information relatingto the source of some or all of the one or several pieces of data and/orgroups of data including, for example, the one or several originatingdevices. In some embodiments, the source database 302 can includeinformation identifying the originating device and/or the user of theoriginating device, the author of some or all of the one or severalpieces of data and/or data groups, previous pieces of data and/or groupsof data originating from the originating device and/or the author, orthe like. In some embodiments, the information stored in the sourcedatabase can be received via the source device 106.

The database server 108 can include an evaluation database 304. Theevaluation database 304 can include information used to direct theevaluation of the pieces of data and/or groups of data. In someembodiments, the evaluation database 304 can contain, for example, theanalysis criteria and the analysis guideline. This information can bereceived from the source device 106, and ultimately from one or severaloriginating devices.

The evaluation database 304 can further include information relating toone or several processing tasks. This information can include, forexample, the identification of independent processing units 114 thathave received one or several pieces of data and/or groups of data forevaluation, the status of the evaluation of one or several pieces ofdata and/or data groups that have been received by an independentprocessing unit 114, an accuracy parameter for one or several of theindependent processing units 114, an evaluation history for the one orseveral independent processing units 114, or the like. In someembodiments, the evaluation criteria can comprise data and/orinstructions in the form of one or several electronic rubrics or scoringguides for use in the evaluation of the one or several pieces dataand/or data groups. The information stored within the evaluationdatabase 304 can be generated within the distributed processing system100, and in some embodiments, this information can be provided to thedistributed processing system 100.

In some embodiments, the evaluation database 304 can include one orseveral evaluations for one or several pieces of data and/or groups ofdata. In some embodiments, these evaluations can include a relativeranking, a characterization, and/or an explanation of thecharacterization and/or relative ranking.

The distributed processing system 100 can include a control database306. The control database 306 can include information used in themanagement of the one or several independent processing units 114. Insome embodiments, this can, for example, include information used inevaluating the quality of the evaluations of the independent processingunits 114, information and/or computer code for generating an accuracyparameter, one or several pieces of learning data and/or learning groupsof data that can provide one or several external stimuli to facilitatelearning of the analysis engine 204, or the like. In some embodiments,these one or several pieces of learning data and/or learning groups ofdata can comprise training materials that are associated with a knowncharacterization and, in some embodiments, can exhibit certain traitsand/or contain certain predetermined material. This training materialcan allow the determination of the quality of the evaluations providedby the one or several independent processing units 114, and particularlyprovided by the analysis modules 204 of the one or several independentprocessing units 114.

With reference now to FIG. 4, a schematic illustration of one embodimentof the server 102 is shown. The server 102 includes a network interface400. The network interface 400 allows the server 102 to access the othercomponents of the distributed processing system 100, and specificallyallows the server 102 to access the network 104 of the distributedprocessing system 100. The server 102 can include features configured tosend and receive information, including, for example, an antenna, amodem, a transmitter, a receiver, or any other feature that can send andreceive information. The network interface 400 can communicate viatelephone, cable, fiber-optic, or any other wired communication network.In some embodiments, the network interface 400 can communicate viacellular networks, WLAN networks, or any other wireless network.

The server 102 can include a user interface 402 that communicatesinformation to, and receives inputs from, a user. The user interface 402can include a screen having one or several pixels, a speaker, a monitor,a keyboard, a microphone, a mouse, a touchpad, a keypad, or any otherfeature or features that can receive inputs from a user and provideinformation to a user. In some embodiments, the pixels of the screen canbe controlled to provide information to the user of the server 102. Insome embodiments, the user interface 402 can allow the user to provideone or several inputs to the server that are relevant to the othercomponents of the distributed processing system 100 discussed herein.

The server 102 can include a distribution engine 404. The distributionengine 404 can be configured to manage the distribution of theprocessing task(s) to the one or several independent processing units114. In one embodiment, the distribution engine can be configured toreceive a processing task comprising one or several pieces of dataand/or data groups from the source device 106, to identify informationrelating to the desired completion of the processing task. In someembodiments, this information can be stored within the analysisguideline, and can, for example, specify a number of evaluations forcompletion of the processing task, a number of repeated evaluations forcompletion of the processing task, one or several attributes ofindependent processing units 114 that can be used for completion of theprocessing task, which can include, for example, a language capability,a subject matter competency level, a training and/or learning level, orthe like. In some embodiments, the distribution engine 404 can receiveinformation relating to one or several of the independent processingunits 114 and can select a set of the independent processing units 114for use in completion of the processing task.

The distribution engine 404 can be further configured to monitor theprogress of the processing task, and can, for example, identify problemsin the completion of the processing task, generate commands and controlportions of the distributed processing system 100 to achieve one orseveral deadlines or to comply with one or several timelines, or thelike.

The server 102 can include a characterization engine 406. Thecharacterization engine 406 can be configured to receive one or severalevaluations from the independent processing units 114 and to generate acharacterization for some or all of the pieces of data and/or groups ofdata included in the processing task. In some embodiments, thecharacterization engine 406 can be configured to receive an accuracyparameter for some or all of the independent processing units 114 thathas generated an evaluation and to use this accuracy parameter, inconnection with the initial characterization contained in the evaluationto generate a final characterization. In some embodiments, this caninclude, for example, retrieving relative ranking information from theevaluation of the one or several pieces of data and/or data groups,generating a ranking for the pieces of data and/or groups of data in theprocessing task, and assigning one or several characterizations based onthe ranking for the pieces of data and/or groups of data in theprocessing task. The characterization engine 406 can be furtherconfigured to identify a discrepancy between the initialcharacterization of the independent processing units 114 and the finalcharacterization. This discrepancy can be compared, directly orindirectly, to the accuracy parameter of the independent processing unit114 that is the source of the initial evaluation, and this discrepancycan be used to update the accuracy parameter of that independentprocessing unit 114.

The server 102 can include a training engine 408. The training engine408 can be configured to generate and/or provide material to one orseveral independent processing units 114 to improve evaluations providedby the independent processing unit 114. In some embodiments, thetraining material can be configured to be used by the analysis engine204 for machine learning by the analysis engine 204. In someembodiments, the training material can be transmitted by the analysisengine 204 to a user of the independent processing unit 114. Thetraining engine 408 can be further configured to track training receivedby one or several of the independent processing units 114 and togenerate a training plan based on, for example, the accuracy parameter,past received training, or the like.

With reference now to FIG. 5, a swim-lane diagram illustrating oneembodiment of a process 500 for increasing the efficiency and processingspeed of distributed computing is shown. The headers of the swim lanesidentify components of the distributed processing system 100 that canperform the indicated step.

The process 500 begins in block 502 wherein the source device 106receives one or several pieces of data and/or data groups. In someembodiments, these can be received from the one or several originatingdevices. After the source device 106 receives the one or several piecesof data and/or data groups, the process 500 proceeds to block 504,wherein the one or several pieces of data and/or data groups areaggregated into a processing task. In some embodiments, this aggregationcan be performed according to information that is received with one orseveral of the one or several pieces of data and/or data groups. Thisinformation can identify a common aspect of the one or several pieces ofdata and/or data groups such as, for example, a type of the one orseveral pieces of data and/or data groups. This information relating tothe common aspect of the one or several pieces of data and/or datagroups can be used to compile the one or several pieces of data and/ordata groups into the processing task. In some embodiments, and as a partof the aggregation of the processing task, the source device 106 canreceive and/or retrieve the analysis criteria and analysis guidelinerelevant to the one or several pieces of data and/or groups of dataforming the processing task. In some embodiments, these can beaggregated into the processing task with the one or several pieces ofdata and/or groups of data. In other embodiments, however, the analysiscriteria and analysis guideline are received and/or retrieved at a laterpoint in process 500.

After the processing task has been aggregated, the process 500 proceedsto block 506, wherein the processing task is sent, from the sourcedevice 106 to the server 102. In some embodiments, the sending of theprocessing task includes any method or technique of moving theprocessing task from the source device 106 to the server 102. In someembodiments, this sending of the processing task can be according to apush-model, and in some embodiments, this sending can be according to apull-model. After the processing task has been sent, the process 500proceeds to block 508, wherein the server 102 receives the processingtask. In some embodiments, the server 102 can receive the processingtask via the network 104 and the network interface 400 of the server102.

After the processing task has been received by the server 102, theprocess 500 proceeds to block 510, wherein a set of independentprocessing units is identified. In some embodiments, this set ofindependent processing units can be identified by evaluating theinformation received from the source device 106, including, for example,the analysis criteria and analysis guideline. In some embodiments, thisset can be identified based on the size of the processing task, based onany deadlines and/or timelines for completion of the processing task, orthe like.

After the set of independent processing units has been identified, theprocess 500 proceeds to block 512, wherein the processing task isseparated into separate pieces of data and/or groups of data, or intogroups of pieces of data and/or groups of data. In some embodiments, theserver 102 can then generate a signal corresponding to the separatepieces of data and/or groups of data, or groups of pieces of data and/orgroups of data, and can, via signal, send the separate pieces of dataand/or groups of data, or groups of pieces of data and/or groups of datato one or several of the independent processing units 114 in the set ofindependent processing units. In some embodiments, this includes anymethod or technique of moving the data from the server 102 to one orseveral of the independent processing units 114. In some embodiments,this can include sending data according to a push-model, and in someembodiments, this can include sending data according to a pull-model.

After the data has been sent, the process 500 proceeds, to block 514,wherein the sent data is received by one of the independent processingunits 114. This data can be received by the independent processing unitvia the network 104 and via the network interface 200. After the datahas been received, the process 500 proceeds to block 516, wherein theanalysis criteria and/or analysis guideline are requested. In someembodiments in which the source device 106 received the analysiscriteria and/or analysis guideline, the intendant processing unit 114can request these from the source device 106. Alternatively, in someembodiments in which the source device 106 received the analysiscriteria and/or analysis guideline, these can be passed to the server102 as part of the processing task, and the server 102 can distributethese to the individual processing units 114.

In the embodiment depicted in FIG. 5, the analysis criteria and/oranalysis guideline are contained in the database server 108, and theindependent processing unit 114 can request these from the databaseserver 108. In some embodiments, this can include the generation of anelectronic query by the independent processing unit 114, which query issent to the database server 108, which database server 108 can receivethe request as shown in step 518. After the database server 108 receivesthe request for the analysis criteria and/or analysis guideline, theprocess 500 proceeds to block 520, wherein the analysis criteria and/oranalysis guideline are sent to the independent processing unit 114. Insome embodiments, this can include the generation of an electricalsignal, which is sent from the database server 108, or the hardwarehousing the database server 108 to the independent processing unit 114.In some embodiments, step 520 can include the receipt of the analysiscriteria and/or analysis guideline by the independent processing unit114.

After the analysis criteria and/or analysis guideline have been sent tothe independent processing unit 114, the process 500 proceeds to block522, wherein the pieces of data and/or groups of data, or groups ofpieces of data and/or groups of data received by the independentprocessing unit 114 at block 514 are analyzed. In some embodiments, thisanalysis can be performed by, at, and/or with the independent processingunit, and specifically, by or with the analysis engine 204. In someembodiments, this can include the updating of the analysis engine 204with the analysis criteria and analysis guideline and the use of theanalysis criteria and analysis guideline to analyze the data. This caninclude, for example, an objective analysis, a subject analysis, asubstantive analysis, a contextual analysis, or the like. The details ofsome embodiments of this analysis will be discussed at greater lengthsbelow.

After the data has been analyzed, the process 500 proceeds to block 524,wherein the analysis result is provided, by the independent processingunit 114 to the server 102. After the analysis result is provided, theprocess 500 proceeds to block 526, wherein the analysis results arereceived by the server 102. In some embodiments, the results can bereceived via one or several electric signals.

After the results have been received, the process 500 proceeds todecision state 528, wherein it is determined if a threshold has beenreached. In some embodiments, this threshold can identify the minimumnumber of evaluations to be performed to complete the processing task.This threshold can, in some embodiments, be specified in the analysisguideline. In some embodiments, the server 102 can track the number ofevaluations that have been performed. In one embodiment, this can bedone by incrementing a value when data is sent for evaluation and/orwhen analysis results are received. In some embodiments, this value canthen identify the number of evaluations performed for a processing task,but can also include information such as, for example, the number oftimes some or all of the one or several pieces of data and/or groups ofdata have been evaluated, which independent processing units 114evaluated which of the one or several pieces of data and/or groups ofdata, or the like.

If it is determined that the threshold has not been reached, then theprocess 500 returns to block 512, and proceeds as outlined above. If itis determined that the threshold has been reached, the process 500proceeds to block 530, wherein the accuracy parameter is received. Insome embodiments, the accuracy parameter can be received from theindependent processing unit 114 at the time that the analysis resultsare received, in some embodiments, the accuracy parameter can bereceived and/or retrieved from the database server 108, and in someembodiments, the accuracy parameter can be generated by the server 102.

After the accuracy parameter has been received, the process 500 proceedsto block 532, wherein the final characterization of some or all of theone or several pieces of data and/or data groups is calculated and/orgenerated. In some embodiments, this final characterization can becalculated and/or generated with the accuracy parameter, as well as withthe evaluation results from one or several independent processing units114. In embodiments in which multiple independent processing units 114generated evaluations for a single piece of data and/or group of data,the calculation of the final characterization can include, for example,adjusting the individual characterizations according to the accuracyvalue, and combining the adjusted values to generate a finalcharacterization.

After the final characterization has been calculated, the process 500proceeds to block 534, wherein the accuracy parameter is updated. Insome embodiments, this can include comparing the final characterizationto some or all of the characterizations of the independent processingunits 114, determining a degree of difference between the two, andgenerating an instance accuracy parameter based on this difference. Insome embodiments, the instance accuracy parameter can be used to adjustthe accuracy parameter; thus if the deviation of a characterization isgreater than indicated by the accuracy parameter, the instance accuracyparameter can be combined with the accuracy parameter to update and/oradjust the accuracy parameter to better reflect this deviation.Conversely, if the deviation of a characterization is less thanindicated by the accuracy parameter, the instance accuracy parameter canbe combined with accuracy parameter to update and/or adjust the accuracyparameter to better reflect this deviation.

With reference now to FIG. 6, a flowchart illustrating one embodiment ofa process 600 for calculating an actual score is shown, which processincreases the efficiency and processing speed of a distributed computingsystem 100. In some embodiments, the process 600 can be performed by thedistributed computing system 100 and/or by a component thereofincluding, for example, the server 102, the independent processing unit114, or any other component of the distributed computing system 100.

The process begins at block 602 wherein the processing task is generatedand/or aggregated. In some embodiments, the processing task can beaggregated at the source device 106, and in some embodiments, theprocessing task can be aggregated at the server 102. In someembodiments, the aggregation of the processing task can include thecombination of one or several pieces of data and/or groups of data.

After the processing task has been generated, the process 600 proceedsto block 604 wherein independent processing unit information isretrieved and/or received. In some embodiments, this information canidentify one or several attributes of one or several independentprocessing units including, for example, one or several attributes ofthe analysis engine 204 and/or the user of the analysis engine. Theseattributes can, for example, identify the number of evaluations providedby one or several of the independent processing units 114 including, forexample, evaluations performed by the analysis engine 204, and/orevaluation performed by a user of the analysis engine 204.

After the independent processing unit information is retrieved, theprocess 600 proceeds to block 606 wherein the processing set isidentified. In some embodiments, the processing set can comprise aplurality of independent processing units and/or user of the independentprocessing units. This set can be selected, in some embodiments, basedon the independent processing unit information that is retrieved inblock 604, the analysis criteria, and/or analysis guideline. In someembodiments, this set can be selected by the server 102 and/or byanother component of the distributed processing system 100.

After the processing set has been identified, the process 600 proceedsto block 608, wherein an independent processing unit 114 is selected. Insome embodiments, the selected independent processing unit 114 can beselected from the processing set. The independent processing unit can,in some embodiments, be selected based on the independent processingunit information that is retrieved in block 604, the analysis criteria,and/or analysis guideline.

After the independent processing unit 114 has been selected, the process600 proceeds to block 610 wherein one or several pieces of data and/ordata groups are encoded and/or converted into an electric signal. Insome embodiments, this can occur at the server 102 and can be part of,for example, the sending of the one or several pieces of data and/orgroups of data to the selected independent processing unit.

After the one or several pieces of data and/or data groups are encodedand/or converted into an electric signal, the process 600 proceeds toblock 612 wherein the one or several pieces of data and/or data groupsare provided. In some embodiments, this step can be performed for all ofthe one or several pieces of data and/or data groups of the processingtask in that some or all of the one or several pieces of data and/ordata groups are sent to an independent processing unit 114. In someembodiments this can result in several independent processing unitsreceiving some of the one or several pieces of data and/or data groups.

In some embodiments, the one or several pieces of data and/or datagroups are provided to the independent processing unit via, for example,the user interface 202. After the one or several pieces of data and/ordata groups have been provided, the process 600 proceeds to block 614wherein the evaluation is received. In some embodiments, the evaluationcan be received from one or several of the independent processing units114. In one embodiment, for example in which the processing task wasdivided into a plurality of pieces of data and/or groups of data, andthen one or several of the plurality of pieces of data and/or groups ofdata was provided to a plurality of independent processing units 114,then the step of block 614 can include receiving the evaluation fromsome or all of this plurality of independent processing units 114. Insome embodiments, the evaluations can be received via, for example, thenetwork 104, and specifically via the user interface 202 of theindependent processing units 114 via the network.

After the evaluation has been received, the process 600 proceeds toblock 616 wherein the evaluation is stored. In some embodiments, forexample, the evaluation can be stored within a portion of thedistributed processing system 100 such as, for example, the databaseserver 108. In one embodiment, the evaluations can be stored in one ofthe databases of the database server 108 such as, for example, theevaluation database 304.

After the evaluation has been stored, the process 600 proceeds to block618 wherein the evaluation guideline is retrieved. In some embodiments,the evaluation retrieved can be retrieved from the database server 108.The evaluation guideline can include a value indicating the number ofevaluations and/or the number of redundant evaluations to be received tocomplete the processing request. In some embodiments, a redundantevaluation can be an additional evaluation of an already evaluated pieceof data and/or group of data.

After the evaluation guideline has been retrieved, the process 600proceeds to block 620 wherein the number of completed evaluations isdetermined. In some embodiments, and as mentioned above, the number orcompleted and/or requested evaluations can be tracked by incrementing avalue when data is sent for evaluation and/or when an evaluation isreceived. In some embodiments, this value can be specific to aprocessing task, and in some embodiments, this value can include aportion that is specific to a processing task, and a portion specific tosome or all of the pieces of data and/or data groups of the processingtask. Thus, in one embodiment, the number of times a piece of dataand/or a group of data is evaluated can be tracked, as well as the totalnumber of evaluations performed for a processing task. In someembodiments, the determination of the number of completed evaluationscan include retrieving the value tracking the number of evaluations.

After the number of completed evaluations is determined, the process 600proceeds to decision state 622 wherein it is determined if theevaluation threshold is achieved. In some embodiments, thisdetermination can be performed by the server 102, and/or anothercomponent of the distributed processing system 100. If it is determinedthat the evaluation threshold has not been achieved, then the process600 returns to block 608 and continues as outlined above.

If it is determined that the evaluation threshold has been achieved,then the process 600 proceeds to block 624 wherein characterizations areretrieved. In some embodiments, this can include retrieving the receivedevaluations, and separating the characterizations from the evaluations.In some embodiments, the evaluations can be retrieved from the databaseserver 108, and the characterizations can be separated from theevaluations by the server 102.

After the characterizations are retrieved, the process 600 proceeds toblock 626 wherein the accuracy parameter is retrieved. In someembodiments, for example, the accuracy parameter can be retrieved fromthe database server 108, and specifically, from one of the databases,such as, for example, the evaluation database 304 and/or the controldatabase 306 of the database server 108.

After the accuracy parameter has been retrieved, the process 600proceeds to block 628 wherein the final characterization, also referredto herein as a combined characterization is generated. In someembodiments, for example, the combined characterization can becalculated by the server 102 of the distributed processing system 100,and can be calculated based on the retrieved characterizations for oneor several of the pieces of data and/or data groups and the accuracyparameters associated with the analysis engine 204, or user thereof, ofthe independent processing units 114 that provided the evaluation. Insome embodiments, some or all of the pieces of data and/or data groupsof the processing task can have characterizations, and a finalcharacterization based on the characterizations and the accuracyparameters associated with the analysis engine 204, or user thereof, ofthe independent processing units 114 that provided the evaluation.

After the actual score has been calculated, the process 600 proceeds toblock 630 wherein the accuracy parameter is updated. In someembodiments, for example, the updating accuracy parameter can includethe comparison of the characterization and the final characterization,the determination of the degree of deviation between thecharacterization and the final characterization, a determination of theupdate and/or augmentation to be provided to the accuracy parameter toreflect the degree of discrepancy between the characterization and thefinal characterization, and a change of the accuracy parameter based onthe update and/or augmentation.

With reference now to FIG. 7, a flowchart illustrating one embodiment ofa process 700 for calculating an accuracy parameter is shown. In someembodiments, the process 700 can be performed by the distributedprocessing system 100 and/or a component thereof including, for example,the server 102.

The process 700 begins at block 702 wherein the independent processingunit 114 is identified. In some embodiments, the independent processingunit 114 can be identified as an independent processing unit 114 thatcompleted an evaluation and/or was included in the set of independentprocessing units. After the independent processing unit 114 has beenidentified, the process 700 proceeds to block 704 wherein the evaluationis retrieved. In some embodiments, the evaluation can be retrieved by acomponent of the distributed processing system 100 such as, for example,the server 102, and can be retrieved, for example, from the databaseserver 104.

After the evaluation has been retrieved, the process 700 proceeds toblock 706 wherein the characterization is retrieved. Thecharacterization can be retrieved from the evaluation retrieved in block704, and/or can be separately retrieved by, for example, the server 102from the database server 108.

After the characterization has been retrieved, the process 700 proceedsto block 708 wherein the final characterization is retrieved. In someembodiments, the final characterization can be retrieved from thedatabase server 108 and/or one of the databases thereof.

After the final characterization has been retrieved, the process 700proceeds to block 710 wherein the final characterization is comparedwith characterization. In some embodiments, this comparison can beperformed by the server 102. After the characterization and the finalcharacterization have been compared, the process 700 proceeds to block712 wherein a difference between the characterization and the finalcharacterization is determined. In some embodiments, this can includedetermining the absolute value and/or magnitude of difference betweenthe characterization and the final characterization; in someembodiments, this can include determining the degree to which thecharacterization score is higher or lower than the finalcharacterization. In some embodiments, this determination can be made bythe server 102.

After the discrepancy between the characterization and the finalcharacterization has been determined, the process 700 proceeds to block714 wherein an index function is retrieved. In some embodiments, theindex function can be used to generate an update value based on thediscrepancy and/or differences between the characterization and thefinal characterization. In some embodiments, for example, the indexfunction can be used to calculate one or several statistical parameterswith respect to one or both of the characterization and the finalcharacterization such as, for example, a standard deviation. After theindex function has been retrieved, the process 700 proceeds to block 716wherein the index function is applied. In some embodiments, for example,the index function can be applied to the server 102. In someembodiments, the application of the index function can include thedetermination of the discrepancy between the characterization and thefinal characterization in terms of the statistical parameters such as,for example, a standard deviation.

After the index function is applied, the process 700 proceeds to block718 wherein the accuracy parameter is updated. In some embodiments, forexample, the update value, which is the result of the applied indexfunction, can be used to augment and/or update the accuracy parameterthat can be, for example, stored within the database server 108.

With reference now to FIG. 8, a flowchart illustrating one embodiment ofa process 800 for analyzing a piece of data and/or a group of data isshown. In some embodiments, this process 800 can be performed at one orseveral of the independent processing units 114, and can be specificallyperformed by the analysis engine 204 and/or with the analysis engine 204by a user.

The process 800 begins at block 802 wherein a piece of data and/or groupof data is received. In some embodiments, this data can be received fromthe server via the network 104 and the network interface 200. After thedata has been received, the process 800 proceeds to block 804, whereinthe analysis criteria and analysis guideline are received. In someembodiments, the analysis criteria and the analysis guideline can bereceived from the server 102, from the source device 106, and/or fromthe database server 108. In some embodiments, the analysis criteria andanalysis guideline can be received simultaneously with the receipt ofthe piece of data and/or the group of data, and in some embodiments, theanalysis criteria and analysis guideline can be received separate fromthe piece of data and/or the group of data. In one embodiment, theindependent processing unit 114 can generate and send a signalrequesting the analysis criteria and/or the analysis guideline, and canreceive the analysis criteria and/or analysis guideline in response tothis signal.

After the analysis criteria and analysis guideline have been received,the process 800 proceeds to block 806, wherein the analysis criteria andanalysis guideline are applied to the received piece of data and/or datagroup. In some embodiments, this step can include identifying one orseveral analysis parameters of the analysis criteria, determining thedegree to which the attributes associated with the analysis parametersoccur, and/or the frequency with which attributes associated with theanalysis parameters occurs in the one or several pieces of data and/orgroups of data. These attributes can relate to, for example, word usage,vocabulary, spelling, grammar, content, citations, use of citations, orthe like. In one embodiment, for example, these criteria can include thenumber and/or frequency of misspellings, the number and/or frequency ofgrammar errors, the level of vocabulary of the one or several pieces ofdata and/or data groups, the correctness and/or level of word usage, thecoherency, accuracy, and/or relevancy of the pieces of data and/or datagroups, or the like. In some embodiments, this step can further includethe application of the analysis guideline to the received piece of dataand/or data group. In some embodiments, the application of the analysiscriteria can result in the generation of one or several parametervalues. In some embodiments, the analysis guideline can influence and/orcontrol how the analysis criteria are applied. This can include, forexample, the degree to which a parameter is evaluated and/or the levelof scrutiny applied to a parameter.

After the analysis criteria and analysis guideline have been applied tothe piece of data and/or data group, the process 800 proceeds to block808, wherein the evaluation is generated. In some embodiments, theevaluation can include information relating to the one or severalparameter values, and how the one or several parameter values werecalculated and/or determined, and in some embodiments, the evaluationcan include a characterization which can be the result of thecombination of the one or several parameter values. In some embodiments,the one or several parameter values can be combined as outlined in theanalysis criteria.

After the evaluation has been generated, the process 800 proceeds toblock 810, wherein the evaluation is provided. In some embodiments, theevaluation can be provided to the server 102 and/or any other componentof the distributed processing system 100. In some embodiments, theproviding of the evaluation can include, for example, the generation ofa signal comprising the evaluation, and the sending of the signal, viathe network 104, to the server 102 or other component of the distributedprocessing system 100.

With reference now to FIG. 9, a flowchart illustrating one embodiment ofa process 900 for generating the evaluation is shown. In someembodiments, the process 900 can be performed in the place of, or as apart of, one or both of blocks 806 and 808 of FIG. 8. The process 900begins at block 902, wherein an analysis parameter is selected. In someembodiments, the analysis parameter can be selected from one of theanalysis parameters of the evaluation criteria. In some embodiments, theselected analysis parameter can be a previous unselected analysisparameter.

After the analysis parameter has been selected, the process 900 proceedsto block 904, wherein one or several manifestations of the attributeassociated with the parameter are identified in the piece of data and/ordata group. In some embodiments, this can be achieved by the review ofthe piece of data and/or data group by the analysis engine and/or userof the analysis engine. The review can include, for example, a review ofspelling, grammar, substance, word usage, vocabulary level, citations,or any other attribute.

After and/or simultaneously with the identification of the one orseveral manifestations of the attribute associated with the parameter inthe piece of data and/or data group, the process 900 proceeds to block906, wherein a parameter value associated with manifestations and/or theanalysis parameter is updated. In some embodiments, this update to thevalue can reflect the frequency of occurrence and/or the degree ofoccurrence of the attribute associated with the analysis parameter inthe piece of data and/or data group.

After the parameter value has been updated, the process 900 proceeds todecision state 908, wherein it is determined if there is an additionalparameter. In some embodiments, this can include, for example,determining whether there are any remaining, unevaluated parameters ofthe analysis criteria. If it is determined that there are additional,unevaluated parameters, then the process 900 returns to block 902 andproceeds as outlined above. If it is determined that there are notadditional, unevaluated parameters, then the process 900 proceeds toblock 910, wherein the parameter values are retrieved. In someembodiments, this can include retrieving all of the parameter values forparameters of the analysis criteria.

After the parameter values have been received, the process 900 proceedsto block 912, wherein the characterization is generated. In someembodiments, the characterization can be generated by the combination ofthe parameter values. These parameter values can be combined by theindependent processing unit 114, and/or by the analysis engine 204 oruser thereof. In some embodiments, these parameter values can becombined according to the analysis guideline.

With reference now to FIG. 10, a flowchart illustrating one embodimentof a process 1000 for parallelized evaluation is shown, whichparallelize evaluation can decrease the amount of time required toprocess an amount of pieces of data and/or data groups. In someembodiments, the process 1000 can be used to determine the relativeranking of one or several pieces of data and/or data groups within aprocessing task and/or to thereby assign and/or determine acharacterization for the one or several pieces of data and/or datagroups. In some embodiments, the process 1000 can provide for theparallel evaluation of a plurality of pieces of data and/or data groupsand a relative ranking of those simultaneously evaluated pieces of dataand/or data groups. The process 1000 can be performed by the distributedprocessing system 100 and/or component thereof including, for example,one or several independent processing units 114, and/or the server 102.

The process 1000 begins at block 1002 wherein two or more pieces of dataand/or data groups are collected. In some embodiments, the pieces ofdata and/or data groups can be collected by the source device 106 from,for example, one or several originating devices. Alternatively, in someembodiments, this collection can be performed by the server 102.

After the work product is collected, the process 1000 proceeds to block1004 wherein the processing task is generated and/or aggregated. In someembodiments, the processing task can be aggregated at the source device106, and in some embodiments, the processing task can be aggregated atthe server 102. In some embodiments, the aggregation of the processingtask can include the combination of one or several pieces of data and/orgroups of data.

After the processing task has been generated, the process 1000 proceedsto block 1006 wherein independent processing unit information isretrieved and/or received. In some embodiments, this information canidentify one or several attributes of one or several independentprocessing units including, for example, one or several attributes ofthe analysis engine 204 and/or the user of the analysis engine. Theseattributes can, for example, identify the number of evaluations providedby one or several of the independent processing units 114 including, forexample, evaluations performed by the analysis engine 204, and/orevaluation performed by a user of the analysis engine 204.

After the independent processing unit information is retrieved, theprocess 1000 proceeds to block 1008 wherein the processing set isidentified. In some embodiments, the processing set can comprise aplurality of independent processing units and/or user of the independentprocessing units. This set can be selected, in some embodiments, basedon the independent processing unit information that is retrieved inblock 1006, the analysis criteria, and/or analysis guideline. In someembodiments, this set can be selected by the server 102 and/or byanother component of the distributed processing system 100.

After the processing set has been identified, the process 1000 proceedsto block 1010, wherein a desired number of comparisons is determined. Insome embodiments, the desired number of comparisons can be defined in,for example, the analysis guideline. In some embodiments, this numbercan vary based on the degree of certainty desired for thecharacterizations and/or the amount of time, cost, and/or processingresources to be consumed by the processing task.

After the desired number of comparisons is determined, the process 1000proceeds to block 1012, wherein one or several pairs of pieces of dataand/or data groups and/or groups of pieces of data and/or data groupsare created. In some embodiments, these groups can be created so that arelative ranking and/or comparison can be performed. Thus, in oneembodiment, these groups can include at least two pieces of data and/ordata groups so that the one can be compared to the other and/orrelatively ranked with respect to the other. In some embodiments, thesepairs can be selected at random, and in some embodiments, these pairscan be selected based on some attribute of the pieces of data and/ordata groups such as, for example, the originator of the pieces of dataand/or data groups, any previous characterizations of pieces of dataand/or data groups from the originator, or the like.

After the one or several pairs of pieces of data and/or data groupsand/or groups of pieces of data and/or data groups have been generated,the process 1000 proceeds to block 1014, wherein the pairs are providedto, for example, one or several independent processing units 114. Insome embodiments, this providing can include the generation and sendingof an electric signal containing the one or several pairs of pieces ofdata and/or data groups and/or groups of pieces of data and/or datagroups. This signal can be generated and/or sent by the server 102.

After the pairs have been provided, the process 1000 proceeds to block1016, wherein the relative rank of one or several pairs of pieces ofdata and/or data groups and/or groups of pieces of data and/or datagroups are received. In some embodiments, these one or several pairs ofpieces of data and/or data groups and/or groups of pieces of data and/ordata groups can be received by the server 102 from one or several of theindependent processing units 114. These relative rankings can identifywhich of the pieces of data and/or data groups of the one or severalpairs and/or groups is (are) relatively better and/or relatively worse.After the relative rank has been received, the process 1000 proceeds toblock 1018, wherein the received relative rank information is stored. Insome embodiments, this information can be stored in, for example, acomponent of the distributed processing system 100 such as, for example,the database server 108.

After the relative rank of the provided work product pieces has beenreceived, the process 1000 proceeds to decision state 1022 wherein it isdetermined if all of the pairs and/or groups in the processing task havebeen ranked. In some embodiments, this determination can be made bycomparison of the number of pairs that were sent to independentprocessing units 114 and the number of evaluations provided byindependent processing units 114. If it is determined that not allpairs/groups have been ranked, then the process can wait a predeterminedperiod and return to block 1016. Alternatively, in some embodiments, ifit is determined that an evaluation of one or several pairs/groups ismissing, the server 102 can identify the independent processing unit 114that received the pairs/groups and send a command/reminder to completethe evaluation and/or send the evaluation.

If it is determined that evaluations have been received for all of thepairs/groups, then the process 1000 proceeds to block 1024 wherein thecount is incremented. In some embodiments, the number of completedand/or requested evaluations can be tracked by incrementing a value whendata is sent for evaluation and/or when an evaluation is received. Insome embodiments, this value can be specific to a processing task, andin some embodiments, this value can include a portion that is specificto a processing task, and a portion specific to some or all of thepieces of data and/or data groups of the processing task. Thus, in oneembodiment, the number of times a piece of data and/or a group of datais evaluated can be tracked, as well as the total number of evaluationsperformed for a processing task. In some embodiments, the determinationof the number of completed evaluations can include retrieving the valuetracking the number of evaluations.

After the count has been incremented, the process 1000 proceeds to block1026 wherein the threshold is retrieved. In some embodiments, thisthreshold can identify the minimum number of evaluations to be performedto complete the processing task. This threshold can, in someembodiments, be specified in the analysis guideline and can be receivedfrom, for example, the database server 108.

After the threshold has been retrieved, the process 1000 proceeds todecision state 1028 wherein it is determined if the threshold has beenachieved. In some embodiment, this can include the comparison of thevalue tracking the number of evaluations to the threshold. Thiscomparison can be performed by the server 102. In some embodiments, aBoolean-value can be assigned to the processing task indicating whetherthe threshold has been achieved. Similarly, in any embodiment describedherein, Boolean values can be assigned to indicate the outcome of adecision state, with a first Boolean value indicating a true conditioncorresponding to a “yes” answer in response to the question of thedecision state and a second Boolean value indicating a false conditioncorresponding to a “no” answer in response to the question of thedecision state. In some embodiments, the server can then use theseBoolean values to identify the outcome of the decision state, and tostore the outcome of the decision state in, for example, the databaseserver 108.

If it is determined that the threshold is not achieved, then the process1000 returns to block 1010 and proceeds as outlined above. If it isdetermined that the threshold has been achieved, then the process 1000proceeds to block 1030 wherein the relative rankings of the pieces ofdata and/or data groups in the pairs/groups are compared to identifyranking discrepancies. In some embodiments, for example, the rankingdiscrepancies can be differences in the relative rankings provided bythe independent processing units 114. In some embodiments, this caninclude the retrieval of the evaluations and/or rankings from theindependent processing units 114 and/or the database server 108 and thecomparison of these evaluations by the server 102.

After the ranking discrepancies have been identified, the process 1000proceeds to block 1032 wherein any identified ranking discrepancies areresolved. In some embodiments, for example, this can include determininga final relative ranking of the pieces of data and/or data groups basedon the provided rankings. In some embodiments, this can include theaveraging of the ranking, and/or the application of a weighting functionto the rankings. This resolution of the rankings of the pieces of dataand/or data groups can be performed by the server 102 and/or any othercomponent of the distributed processing system 100.

After the ranking discrepancies have been resolved, the process 1000proceeds to block 1034 wherein the processing task is ranked. In someembodiments, this can include applying the final rankings, including,for example, any resolved rankings to the processing task, andparticularly to the pieces of data and/or data groups forming theprocessing task. In some embodiments, this can be performed by theserver 102 and/or any other component of the distributed processingsystem 100.

After the processing task is ranked, the process 1000 proceeds to block1036 wherein one or several bounding data pieces are identified. In someembodiments, bounding data pieces can be pieces of data and/or groups ofdata located between characterization levels such that all the pieces ofdata and/or data groups that are relatively higher ranked than thebounding piece have a characterization from a first subset of scoresthat is higher than the characterization of the bounding piece, and allthe pieces of data and/or data groups that are ranked lower than thebounding piece have a characterization from a second subset of scoresthat is lower than the characterization of the bounding piece.

In some embodiments, the identification of the bounding pieces caninclude selecting one or several pieces of data and/or data groups ofthe processing task as potential bounding pieces, characterizing the oneor several pieces of data and/or data groups of the processing, andselecting one or several of the potential bounding pieces as boundingpieces. In some embodiments, the one or several potential boundingpieces can be selected based on one or several parameters and/orattributes of the pieces of data and/or data groups of the processingtask. In one embodiment, for example, the bounding pieces can beselected based on, for example, the past characterizations of one orseveral pieces of data and/or data groups from one or several of theoriginating devices. In some embodiments, the one or several boundingpieces can be selected by performing a brief analysis of the pieces ofdata and/or data groups in the processing task.

After the potential bounding pieces have been selected, and as mentionedabove, a characterization can be given to some or all of the potentialbounding pieces. This characterization can proceed according to theprocess 900 depicted in FIG. 9. After the characterizations of thepotential bounding pieces have been received, and as mentioned above,one or several bounding pieces can be selected. In some embodiments, theone or several selected bounding pieces can, as described above, have acharacterization that is between levels.

Alternatively to the above steps for identifying one or several boundingpieces, in some embodiments, the bounding pieces can be one or severalpre-generated pieces of data and/or data groups that can be, forexample, created so as to have a desired characterization. In such anembodiment, the step of block 1036 can comprise retrieving thesebounding pieces from, for example, the database server 108.

After the bounding pieces have been identified, the process 1000proceeds to block 1040 wherein the processing task is characterized. Insome embodiments, the characterization of the processing task can beachieved by identifying the relative rank of the one or several boundingpieces within the processing task. In some embodiments, in which thebounding pieces are pre-generated pieces of data and/or data groups,this can include performing the steps of blocks 1010 through 1018 todetermine a relative rank of the bounding pieces. After the boundingpieces are relatively ranked, or in some embodiments in which thebounding pieces are identified from the pieces of data and/or groups ofdata of the processing task, the application of the characterization tothe processing task can include identifying pieces of data relativelypositioned between two adjacent bounding pieces, and assigningcharacterization based on these bounding pieces. Thus, when the boundingpieces are selected such that they are between characterization levels,pieces of data and/or data groups that are relatively bounded by twobounding pieces can be assigned the characterization level that islikewise bounded by the two bounding pieces. In other embodiments, thecharacterization level can be determined based on the separation ofcharacterization levels between the bounding scores, and a determinationof the degree of separation between the adjacent bounding scores and anintermediate piece of data and/or data group. This degree of separationcan be expressed as a fraction having a denominator of the total numberof pieces of data and/or data groups between the bounding pieces, and anumerator of the number of pieces of data and/or data groups separatingthe intermediate piece of data from one of the bounding pieces. In suchan embodiment, the characterization level can be calculated as thecharacterization level of the bounding piece having a relatively lowercharacterization level plus the value created by the multiplication ofthe fractional representation of the degree of separation and thedifference in the characterization levels of the bounding pieces.Embodiments can be performed by the server 102 and/or any othercomponent of the distributed processing system 100.

With reference now to FIG. 11, a flowchart illustrating one embodimentof a process 1100 for analyzing a piece of data and/or a group of datais shown. In some embodiments, this process 1100 can be performed at oneor several of the independent processing units 114, and can bespecifically performed by the analysis engine 204 and/or with theanalysis engine 204 by a user.

The process 1100 begins at block 1102 wherein a pair/group is received.In some embodiments, the pair/group can comprise two or more pieces ofdata and/or data groups. In some embodiments, this pair/group can bereceived from the server 102 via the network 104 and the networkinterface 200. After the pair/group has been received, the process 1100proceeds to block 1104, wherein the analysis criteria and analysisguideline are received. In some embodiments, the analysis criteria andthe analysis guideline can be received from the server 102, from thesource device 106, and/or from the database server 108. In someembodiments, the analysis criteria and analysis guideline can bereceived simultaneous with the receipt of the pair/group, and in someembodiments, the analysis criteria and analysis guideline can bereceived separate from the pair/group. In one embodiment, theindependent processing unit 114 can generate and send a signalrequesting the analysis criteria and/or the analysis guideline, and canreceive the analysis criteria and/or analysis guideline in response tothis signal.

After the analysis criteria and analysis guideline have been received,the process 1100 proceeds to block 1106, wherein the analysis criteriaand analysis guideline are applied to the received pair/group. In someembodiments, this step can include identifying one or several analysisparameters of the analysis criteria, determining the relative rank ofthe pair/group with respect to these one or several analysis parameters,which can be based on, for example, the relative degree to which theattributes associated with the analysis parameters occur, and/or therelative frequency with which attributes associated with the analysisparameters occurs in the one or several pieces of data and/or groups ofdata in the pair/group. These attributes can relate to, for example,word usage, vocabulary, spelling, grammar, content, citations, use ofcitations, or the like. In one embodiment, for example, these criteriacan include the number and/or frequency of misspellings, the numberand/or frequency of grammar errors; the level of vocabulary of the oneor several pieces of data and/or data groups; the correctness and/orlevel of word usage; the coherency, accuracy, and/or relevancy of thepieces of data and/or data groups; or the like. In some embodiments,this step can further include the application of the analysis guidelineto the received pair/group. In some embodiments, the application of theanalysis criteria can result in the relative ranking of the pieces ofdata and/or data groups of the pair with respect to one or severalanalysis parameters.

After the analysis criteria and analysis guideline have been applied topair/group, the process 1100 proceeds to block 1108, wherein therelative ranking is generated. In some embodiments, the relativerankings with respect to the one or several analysis parameters can becombined to create an overall relative ranking. In some embodiments, theone or several relative rankings with respect to the analysis parameterscan be combined as outlined in the analysis criteria.

After the evaluation has been generated, the process 1100 proceeds toblock 1110, wherein the relative ranking is provided. In someembodiments, the relative ranking can be provided to the server 102and/or any other component of the distributed processing system 100. Insome embodiments, the providing of the relative ranking can include, forexample, the generation of a signal comprising the relative ranking, andthe sending of the signal, via the network 104, to the server 102 orother component of the distributed processing system 100.

With reference now to FIG. 12, a flowchart illustrating one embodimentof a process 1200 for generating the relative ranking is shown. In someembodiments, the process 1200 can be performed in the place of, or as apart of, one or both of blocks 1106 and 1108 of FIG. 11. The process1200 begins at block 1202, wherein an analysis parameter is selected. Insome embodiments, the analysis parameter can be selected from one of theanalysis parameters of the evaluation criteria. In some embodiments, theselected analysis parameter can be a previously unselected analysisparameter.

After the analysis parameter has been selected, the process 1200proceeds to block 1204, wherein one or several manifestations of theattribute associated with the parameter are identified in the pieces ofdata and/or data groups of the pair/group. In some embodiments, this canbe achieved by the review of the pieces of data and/or data groups ofthe pair/group by the analysis engine 204 and/or user of the analysisengine 204. The review can include, for example, a review of spelling,grammar, substance, word usage, vocabulary level, citations, or anyother attribute.

After and/or simultaneously with the identification of the one orseveral manifestations of the attribute associated with the parameter inthe pieces of data and/or data groups of the pair/group, the process1200 proceeds to block 1206, wherein a parameter relative ranking iscreated, and particularly wherein a parameter relative ranking valueidentifying the relative degree to which the attributes associated withthe analysis parameters occur, and/or the relative frequency with whichattributes associated with the analysis parameters occurs in the one orseveral pieces of data and/or groups of data of the pair/group iscreated. In some embodiments, this can include comparing the relativedegree to which the attributes associated with the analysis parametersoccur, and/or the relative frequency with which attributes associatedwith the analysis parameters occurs in the one or several pieces of dataand/or groups of data of the pair/group, and identifying which of thepieces of data and/or data groups is relatively better with regard tothat analysis parameter. In some embodiments, a first value can beassociated with a piece of data and/or data group if it is relativelybetter with respect to an analysis parameter, a second value can beassociated with a piece of data and/or data group if it is relativelyworse with respect to an analysis parameter, and a third value can beassociated with a piece of data and/or data group if it is relativelyequal with respect to an analysis parameter.

After the parameter relative ranking has been created, the process 1200proceeds to decision state 1208, wherein it is determined if there is anadditional parameter. In some embodiments, this can include, forexample, determining whether there are any remaining, unevaluatedparameters of the analysis criteria. If it is determined that there areadditional, unevaluated parameters, then the process 1200 returns toblock 1202 and proceeds as outlined above. If it is determined thatthere are not additional, unevaluated analysis parameters, then theprocess 1200 proceeds to block 1210, wherein the parameter relativeranking values are retrieved. In some embodiments, this can includeretrieving all of the parameter relative ranking values for parametersof the analysis criteria.

After the parameter relative ranking values have been received, theprocess 1200 proceeds to block 1212, wherein the relative ranking of thepair/group is generated. In some embodiments, the relative ranking canbe generated by the combination of the parameter relative rankings. Insome embodiments, these parameter relative ranking values can becombined according to the analysis guideline. In some embodiments, forexample, some or all of the analysis parameters can be associated with aweighting value. In such an embodiment, the combined relative rankingcan be calculated by a combination of the weighting values and theparameter relative ranking values. In one embodiment, for example, someor all of the parameter ranking values can be multiplied by theirweighting value, and the products of those multiplications can be addedto create a combined value. In one such embodiment, the combined valuesof the pieces of data and/or data groups of the pair/group can becompared to determine the relative ranking of the pair/group, asindicated in block 1214 of FIG. 12. Thus, in one embodiment, a piece ofdata and/or data group can be relatively better if its combined value islarger than the combined value of another piece of data and/or datagroup. These parameter relative rankings can be generated by theindependent processing unit 114, and/or by the analysis engine 204 oruser thereof.

With reference now to FIG. 13, an exemplary environment with whichembodiments may be implemented is shown with a computer system 1300 thatcan be used by a user 1304 as all or a component of a crowd-scoringsystem 100. The computer system 1300 can include a computer 1302,keyboard 1322, a network router 1312, a printer 1308, and a monitor1306. The monitor 1306, processor 1302 and keyboard 1322 are part of acomputer system 1326, which can be a laptop computer, desktop computer,handheld computer, mainframe computer, etc. The monitor 1306 can be aCRT, flat screen, etc.

A user 1304 can input commands into the computer 1302 using variousinput devices, such as a mouse, keyboard 1322, track ball, touch screen,etc. If the computer system 1300 comprises a mainframe, a designer 1304can access the computer 1302 using, for example, a terminal or terminalinterface. Additionally, the computer system 1326 may be connected to aprinter 1308 and a server 1310 using a network router 1312, which mayconnect to the Internet 1318 or a WAN.

The server 1310 may, for example, be used to store additional softwareprograms and data. In one embodiment, software implementing the systemsand methods described herein can be stored on a storage medium in theserver 1310. Thus, the software can be run from the storage medium inthe server 1310. In another embodiment, software implementing thesystems and methods described herein can be stored on a storage mediumin the computer 1302. Thus, the software can be run from the storagemedium in the computer system 1326. Therefore, in this embodiment, thesoftware can be used whether or not computer 1302 is connected tonetwork router 1312. Printer 1308 may be connected directly to computer1302, in which case, the computer system 1326 can print whether or notit is connected to network router 1312.

With reference to FIG. 14, an embodiment of a special-purpose computersystem 1404 is shown. The above methods may be implemented bycomputer-program products that direct a computer system to perform theactions of the above-described methods and components. Each suchcomputer-program product may comprise sets of instructions (codes)embodied on a computer-readable medium that directs the processor of acomputer system to perform corresponding actions. The instructions maybe configured to run in sequential order, or in parallel (such as underdifferent processing threads), or in a combination thereof. Afterloading the computer-program products on a general purpose computersystem 1326, it is transformed into the special-purpose computer system1404.

Special-purpose computer system 1404 comprises a computer 1302, amonitor 1306 coupled to computer 1302, one or more additional useroutput devices 1430 (optional) coupled to computer 1302, one or moreuser input devices 1440 (e.g., keyboard, mouse, track ball, touchscreen) coupled to computer 1302, an optional communications interface1450 coupled to computer 1302, a computer-program product 1405 stored ina tangible computer-readable memory in computer 1302. Computer-programproduct 1405 directs system 1404 to perform the above-described methods.Computer 1302 may include one or more processors 1460 that communicatewith a number of peripheral devices via a bus subsystem 1490. Theseperipheral devices may include user output device(s) 1430, user inputdevice(s) 1440, communications interface 1450, and a storage subsystem,such as random access memory (RAM) 1470 and non-volatile storage drive1480 (e.g., disk drive, optical drive, solid state drive), which areforms of tangible computer-readable memory.

Computer-program product 1405 may be stored in non-volatile storagedrive 1480 or another computer-readable medium accessible to computer1302 and loaded into memory 1470. Each processor 1460 may comprise amicroprocessor, such as a microprocessor from Intel® or Advanced MicroDevices, Inc.®, or the like. To support computer-program product 1405,the computer 1302 runs an operating system that handles thecommunications of product 1405 with the above-noted components, as wellas the communications between the above-noted components in support ofthe computer-program product 1405. Exemplary operating systems includeWindows® or the like from Microsoft® Corporation, Solaris® from Oracle®,LINUX, UNIX, and the like.

User input devices 1440 include all possible types of devices andmechanisms to input information to computer system 1302. These mayinclude a keyboard, a keypad, a mouse, a scanner, a digital drawing pad,a touch screen incorporated into the display, audio input devices suchas voice recognition systems, microphones, and other types of inputdevices. In various embodiments, user input devices 1440 are typicallyembodied as a computer mouse, a trackball, a track pad, a joystick,wireless remote, a drawing tablet, a voice command system. User inputdevices 1440 typically allow a user to select objects, icons, text andthe like that appear on the monitor 1306 via a command such as a clickof a button or the like. User output devices 1430 include all possibletypes of devices and mechanisms to output information from computer1302. These may include a display (e.g., monitor 1306), printers,non-visual displays such as audio output devices, etc.

Communications interface 1450 provides an interface to othercommunication networks 1495 and devices and may serve as an interface toreceive data from and transmit data to other systems, WANs and/or theInternet 1318. Embodiments of communications interface 1450 typicallyinclude an Ethernet card, a modem (telephone, satellite, cable, ISDN), a(asynchronous) digital subscriber line (DSL) unit, a FireWire®interface, a USB® interface, a wireless network adapter, and the like.For example, communications interface 1450 may be coupled to a computernetwork, to a FireWire® bus, or the like. In other embodiments,communications interface 1450 may be physically integrated on themotherboard of computer 1302, and/or may be a software program, or thelike.

RAM 1470 and non-volatile storage drive 1480 are examples of tangiblecomputer-readable media configured to store data such ascomputer-program product embodiments of the present invention, includingexecutable computer code, human-readable code, or the like. Other typesof tangible computer-readable media include floppy disks, removable harddisks, optical storage media such as CD-ROMs, DVDs, bar codes,semiconductor memories such as flash memories, read-only-memories(ROMs), battery-backed volatile memories, networked storage devices, andthe like. RAM 1470 and non-volatile storage drive 1480 may be configuredto store the basic programming and data constructs that provide thefunctionality of various embodiments of the present invention, asdescribed above.

Software instruction sets that provide the functionality of the presentinvention may be stored in RAM 1470 and non-volatile storage drive 1480.These instruction sets or code may be executed by the processor(s) 1460.RAM 1470 and non-volatile storage drive 1480 may also provide arepository to store data and data structures used in accordance with thepresent invention. RAM 1470 and non-volatile storage drive 1480 mayinclude a number of memories including a main random access memory (RAM)to store of instructions and data during program execution and aread-only memory (ROM) in which fixed instructions are stored. RAM 1470and non-volatile storage drive 1480 may include a file storage subsystemproviding persistent (non-volatile) storage of program and/or datafiles. RAM 1470 and non-volatile storage drive 1480 may also includeremovable storage systems, such as removable flash memory.

Bus subsystem 1490 provides a mechanism to allow the various componentsand subsystems of computer 1302 communicate with each other as intended.Although bus subsystem 1490 is shown schematically as a single bus,alternative embodiments of the bus subsystem may utilize multiple bussesor communication paths within the computer 1302.

A number of variations and modifications of the disclosed embodimentscan also be used. Specific details are given in the above description toprovide a thorough understanding of the embodiments. However, it isunderstood that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Implementation of the techniques, blocks, steps and means describedabove may be done in various ways. For example, these techniques,blocks, steps and means may be implemented in hardware, software, or acombination thereof. For a hardware implementation, the processing unitsmay be implemented within one or more application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, other electronic units designed toperform the functions described above, and/or a combination thereof.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a swim diagram, a dataflow diagram, a structure diagram, or a block diagram. Although adepiction may describe the operations as a sequential process, many ofthe operations can be performed in parallel or concurrently. Inaddition, the order of the operations may be re-arranged. A process isterminated when its operations are completed, but could have additionalsteps not included in the figure. A process may correspond to a method,a function, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a function, its termination corresponds to a return ofthe function to the calling function or the main function.

Furthermore, embodiments may be implemented by hardware, software,scripting languages, firmware, middleware, microcode, hardwaredescription languages, and/or any combination thereof. When implementedin software, firmware, middleware, scripting language, and/or microcode,the program code or code segments to perform the necessary tasks may bestored in a machine readable medium such as a storage medium. A codesegment or machine-executable instruction may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a script, a class, or any combination of instructions,data structures, and/or program statements. A code segment may becoupled to another code segment or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, and/or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

For a firmware and/or software implementation, the methodologies may beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine-readable mediumtangibly embodying instructions may be used in implementing themethodologies described herein. For example, software codes may bestored in a memory. Memory may be implemented within the processor orexternal to the processor. As used herein the term “memory” refers toany type of long term, short term, volatile, nonvolatile, or otherstorage medium and is not to be limited to any particular type of memoryor number of memories, or type of media upon which memory is stored.

Moreover, as disclosed herein, the term “storage medium” may representone or more memories for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine readable mediums for storing information. The term“machine-readable medium” includes, but is not limited to portable orfixed storage devices, optical storage devices, and/or various otherstorage mediums capable of storing that contain or carry instruction(s)and/or data.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the disclosure.

What is claimed is:
 1. A distributed processing method for evaluatingstudent work product, the method comprising: receiving at at least oneserver a signal encoding a processing task; separating a one or severalgroups of data from the processing task; sending a first one of one orseveral groups of data to a first evaluator device and to a secondevaluator device; receiving a first score for the first one of one orseveral groups of data from the first evaluator device; receiving asecond score for the first one of the one or several groups of data fromthe second evaluator device; retrieving an accuracy parameter comprisinga first component for standardizing the first score and a secondcomponent for standardizing the second score; calculating a combinedscore of the first one of the one or several groups of data based on acombination of the first score, the second score, and the accuracyparameter; and updating the accuracy parameter, wherein updating theaccuracy parameter comprises updating the first component of theaccuracy parameter and generating an updated accuracy parameter.
 2. Themethod of claim 1, wherein the at least one server is further configuredto select a group of evaluators to process the processing task.
 3. Themethod of claim 2, wherein the at least one server comprises a storedmanagement guideline comprising one or several rules for a completion ofan analysis of the processing task.
 4. The method of claim 3, whereinthe stored management guideline specifies a number of reviews for one ormore of the one or several groups of data in the processing task.
 5. Themethod of claim 1, wherein at least one of the evaluator devicescomprises a cloud computing device.
 6. The method of claim 1, whereinthe at least one server further performs the actions of: incrementing acount when the first one of the one or several groups of data is sent toone of the evaluator devices; and comparing the count to a thresholdvalue to determine whether to send the first one of the one or severalgroups of data to an additional one of the plurality of evaluators. 7.The method of claim 6, wherein the at least one server further performsthe actions of selecting an additional one of the plurality ofevaluators after determining to send the first one of the one or severalgroups of data to an additional one of the plurality of evaluators. 8.The method of claim 7, wherein the at least one server further performsthe actions of receiving an additional electric signal from theadditional one of the evaluator devices, and wherein the additionalelectric signal comprises an additional score generated by theadditional one of the evaluator devices of the first one of the one orseveral groups of data.
 9. The method of claim 1, wherein the at leastone server further performs the actions of updating the accuracyparameter with respect to the first one of the evaluator devices,wherein the updating compares the combined score of the first one of theone or several groups of data to the first score and the accuracyparameter to generate a result, and incrementing or decrementing thefirst component of the accuracy parameter based on result.
 10. Themethod of claim 1, wherein updating the accuracy parameter furthercomprises retrieving an index function, wherein the index functiongenerates an accuracy parameter update value for the first component ofthe accuracy parameter according to a difference between the first scoreand the combined score.
 11. The method of claim 10, wherein the firstcomponent of the accuracy parameter is incremented or decremented by theaccuracy parameter update value.
 12. The method of claim 1, wherein theat least one server further performs the actions of: selecting one orseveral of the evaluator devices for scoring of a potential boundingpiece, wherein one of the several of the evaluators are selected basedon an accuracy parameter retrieved by the at least one server, whereinthe accuracy parameter identifies a degree to which one or several ofthe evaluators have accurately scored a one or several previous datagroups; and calculating a score of the one of the one or several groupsof data of the processing task based on the potential bounding piece andthe accuracy parameter.
 13. The method of claim 12, wherein thepotential bounding piece comprises one of the one or several groups ofdata and an associated score.
 14. The method of claim 13, wherein the atleast one server further performs the actions of retrieving thepotential bounding piece from memory.
 15. The method of claim 13,wherein the at least one server further performs the actions ofcontrolling a generation of the potential bounding piece.
 16. The methodof claim 15, wherein the at least one server further performs theactions of selecting a selected one or several of the evaluators forscoring of a potential bounding piece, wherein the one or several of theevaluators are selected based on an accuracy parameter retrieved by theat least one server, wherein the accuracy parameter identifies thedegree to which one or several of the evaluators have accurately scoredone or several previous data groups.
 17. The method of claim 16, whereinthe at least one server further performs the actions of: identifying oneof the one or several groups of data as the potential bounding piece;generating a bounded signal, wherein the bounded signal encodes thepotential bounding piece; sending the bounded signal to the selected oneor several of the evaluator devices; receiving a response signal fromthe selected one or several of the evaluator devices, wherein theresponse signal comprises a bounded score of the potential boundingpiece generated by the selected one or several of the evaluator devices;and storing an indicator identifying the potential bounding piece as abounding piece and storing the bounded score of the potential boundingpiece.
 18. The method of claim 12, wherein the potential bounding piececomprises a plurality of bounding pieces, wherein some of the pluralityof bounding pieces comprise different bounded scores.
 19. The method ofclaim 12, wherein at least one of the evaluator devices comprises amobile computing device.
 20. The method of claim 12, wherein thepotential bounding piece comprises a plurality of bounding pieces,wherein all of the plurality of bounding pieces comprise differentbounded scores.